Approaches used by Faculty to Assess Critical Thinking - Implications for General Education 


Mark Nicholas - Oklahoma State University 
Miriam Raider-Roth - University of Cincinnati 


Prepared for ASHE Annual Conference, 2011 



1 


Abstract 

This investigation focused on a group of 17 faculty drawn from disciplines in the humanities 
social sciences and natural sciences. Using in-depth interviews, focus group discussions and 
qucditative coding strategies, this study examined how faculty conceptualized the term critical thinking 
(CT), and how they assessed for it in general education courses. The study adopted an exploratory 
approach and hence did not use an operational definition of CT. 

This study found that that faculty assessed for CT implicitly through disciplinary content. Their 
approach focused on aspects of CT that were most relevant to their disciplinary needs. Faculty in the 
natural sciences leaned toward rationalistic, empirical approaches, faculty from philosophy focused 
on rationality and formulaic procedurcd processes, faculty in the humanities focused on aporetic 
elements of CT, and faculty in the social sciences relied on empirical artifacts and standards while 
recognizing the role of perspective in CT. Faculty rejected the use of multiple choice tests as a valid 
means of assessing CT. They assessed for CT through other communicative learning outcomes. 

Faculty did not use formed assessment instruments to explicitly assess for CT and hence had no way to 
gauge the efficacy of their pedagogical approaches. 

This study raises questions on the efficacy of discipline -genered assessment approaches used by 
national governing bodies, commercial standardized tests and genered education programs to teach 
and assess CT in general education. It presents empirical evidence on the need for a complex, multi- 
disciplinary, inclusive approach to conceptualizing CT. An approach that reflects how faculty 
approach CT in the undergraduate classroom and fulfills the aspirations for multi-clisciplinarity in 
genered education. The findings of this study have implications for the assessment of CT in genered 
education, national accountability efforts and for faculty development. 

Key Words: critical thinking, faculty, assessment, genered education 
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Introduction 

The Spellings Report (USDE, 2006) brought the spotlight of accountability to higher education 
just as No Child Left Behind brought accountability to PK-12 education. The Voluntary System of 
Accountability (VSA) which was the response of 4-year universities to the Spelling’s Report called on 
its participants to voluntarily disclose “clear, accessible and comparable” (VSA, 2011) gains in critical 
thinking (CT) among other student learning outcomes to circumvent direct government intervention. 
Under the system, universities are to use a common reporting method called College Portrait and 
publish the results of institutional student learning outcomes on the university’s website (NASULGC, 
2007). 

The VSA recommended that institutions use one of three standardized tests to achieve 
institutional comparability in how CT is assessed and reported. This issue is being hotly debated in 
assessment circles; nonetheless, it is being implemented (VSA, 2008). In a related study, Escoe, Hall, 
and Nicholas (2009) compared two contrasting methods that assessed for discipline-general CT using 
the same sample of students. The first was the Collegiate Learning Assessment (CLA), a VSA 
recommended, standardized, 90 minute test. The other method applied a faculty developed, discipline- 
general CT rubric sponsored by the AAC&U. The rubric was applied by faculty to student artifacts that 
emerged from the classroom. The study found little correlation at the .05 level between the results of 
faculty driven assessment and the CLA when assessing CT. 

This is an intriguing finding when one considers that the outcomes of assessments developed 
and applied by faculty did not match with an approach recommended by the VSA to hold institutions 
accountable for the development of CT. Nicholas, Szabo, and Weber (2009) pointed out that CT as a 
term, is used by multiple stakeholders in higher education and argued that each player brings a unique 
dimension; and some, a vested interest to the term. Hence it is important to decode and deconstruct the 
term CT and to establish meaning and connections within the multiple contexts in which CT operates 


in higher education. 



3 


While much of the dust kicked up by the Spellings Report has settled, the specter of 
accountability still looms over higher education. In addition, the response of the VS A with regard to 
CT in many ways concretized some of the dominant discourses on CT. It also brought to the forefront 
some unexamined assumptions on which general education in the United States is based. Thereby 
providing a good opportunity to re-examine such assumptions and discourses on CT in light of how 
faculty approach the assessment of CT in the classroom. 

Discourses on CT in Higher Education 

This research endeavor is situated within scholarly discourses in several bodies of literature. In 
order to capture the complexity of critical thinking (CT) as a concept, the diversity of views on it and 
the multiple contexts in which it operates in higher education, we examine discourses on CT as a 
concept and discourses on the application of CT within higher education and general education 
programs in particular. We will also identify areas in which further understanding is needed, and 
establish a framework for the current study. 

CT as a Concept 

An analysis of the research literature on CT revealed that there are multiple definitions, varied 
conceptualizations, multiple taxonomies of skills and dispositions, numerous instructional methods, 
and competing assessment measures. Views on CT range across a spectrum from it being a purely 
cognitive, rational skill that helps establish truth (Ennis, 2007) to it belonging to the affective, 
dispositional and behavioral domains, where context, emotions and other subjective elements play a 
role in CT (Tishman & Andrade, 1996). 

Papastephanou and Charoula (2007) identified two dominant conceptualizations of CT - the 
rationalistic and the skilled approach. They proposed a third alternative which they described as the 
critical philosophical approach to CT. As a rationalistic skill, CT is defined as a collection of cognitive 
skills that focus heavily on rationality (Ennis, 1985; Glaser, 1985; Swartz, 1987; Elder & Paul, 2004). 
The rationalistic approach focused on empiricism as the means to reach a desired outcome as a result 
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of the CT process. The skilled approach focused on the CT as the use of formal logic and deductive 
processes. The skilled and rationalistic approaches in combination treat CT as a generic and 
independent skill and can be taught outside disciplinary contexts (Moore 2004, 2011). A logical 
outflow of using the rationalistic and skilled approach is that CT can be assessed on a multiple choice 
test, focusing on a set of discipline-general, rationalistic skills. The conceptualization that underlies the 
tests that subscribe to this school, focus heavily on rationalistic, measurable aspects of CT like logic, 
analysis, inference and deduction that lead to a desired conclusion. Describing such approaches, 
Sternberg, Penn and Hawkins (2011, in press) questioned, “this model of measurement may be 
appropriate for assessing the analytical aspects of undergraduate education, but is it as appropriate for 
assessing other skills that may matter more later in life. . .?” 

The definitions from the tradition of critical philosophy focus on multiple ways of knowing 
with a goal to deconstruct established principles and current ways of knowing. Garside (1996) defined 
CT as “a controlled sense of skepticism”. Halx and Reybold (2005) concluded that the definitions in 
this tradition focus on the uncertainty of knowledge and knowing. Generally, aporetic facets of CT like 
questioning, wondering, comfort with ambiguity, irreducible complexity and an appreciation for the 
inconclusive nature of knowledge are valued. These elements do not figure in standardized or 
discipline-general approaches to CT. 

CT and University Curricular, Pedagogical and Assessment Approaches 

CT is traditionally taught to undergraduate students as a general education requirement. The 
general education model has adopted what Ennis (1989) called the “immersion approach” (p. 4) in 
which students are immersed into the subject matter thoughtfully but CT is taught implicitly. 
Commercial instruments like the Watson and Glaser Test, The Cornell Critical Thinking Test and the 
California Critical Thinking Skills Test which are used extensively in research studies also measure 


discipline-general CT. 
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There are several implications to assessing CT as discipline-general. First, it draws heavily 
from the dominant rationalistic, skilled based conceptualization of CT which is also one of the 
narrowest. Second, it operates under the assumption that a standard set of CT skills can be assessed in 
all undergraduate students irrespective of discipline. Under that assumption, a student who is a cello 
major in a conservatory of music and a student of engineering or sociology can be assessed for CT on 
the same standardized test. Third, it means that institutions will only be held accountable for the 
development of a narrow interpretation of CT in all baccalaureate students; defeating the true essence 
of CT. Fourth, there is little research that examines the discipline general assumption with the realities 
with how faculty approach the assessment of CT in the classroom. 

Faculty play a key role in making a conceptualization of CT operational within the classroom 
(Tsui, 2001). Halx and Reybold (2005) found that research into faculty perceptions of CT had received 
little attention. Important questions that remain unanswered are how do faculty conceptualize CT? Are 
such conceptualizations different across disciplines? Do faculty also take a discipline-general approach 
to teaching and assessing for CT as does the dominant assessment approach? These are pertinent 
questions to ask if we are going to hold faculty and institutions accountable based on the outcomes of 
discipline-general assessment tests and measures. They are important questions to ask if we are to 
retain the assumption in higher education that CT is discipline-general. They are essential questions 
that can throw light on why faculty approaches to assessing CT do not match the results of 
standardized, discipline-general tests of CT. 

Yet, empirical research on how faculty from different disciplines conceptualized or assessed for 
CT is scarce. The few that exist are evaluative, or treat faculty as a bloc and do not take a 
multidisciplinary approach. Zygmont and Schaefer (2006) concluded that the problems with measuring 
CT may actually lie in disciplinary influences on CT. Penn (201 1) wrote that there is need for research 
that explored the relationship between general education outcomes like CT and the disciplines. 
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In an attempt to present empirical data on these unanswered questions, this study examined 
how faculty approached the assessment of CT. The study adopted a strong disciplinary approach 
focusing on disciplines in the humanities, social sciences and natural sciences. It studied faculty 
conceptualizations in light of the approach that they adopted to assessment. It provides valuable 
insights that can help examine if and how faculty approaches are aligned with models used at the 
institutional and national levels. The study used qualitative, inductive and interpretivist approaches to 
study CT within the context of how faculty assessed for it. The study took an exploratory approach and 
hence has not used an operational definition of CT. 

Research Procedures 

Research Location 

This study was conducted at two large public universities. The first university was located in 
the Midwest and enrolled around 36,000 undergraduate students. The other was in the Southwest and 
enrolled around 20,000 undergraduate students. These universities will be referred to using 
pseudonyms - Mid-Western University and South-Western University. Both universities shared 
similarities that are relevant to this study. 

Both universities are public, land-grant universities and are part of the Voluntary System of 
Accountability (VSA) for reporting general education outcomes like CT using College Portrait ®. The 
Carnegie Foundation had classified both universities as Research One and their general education 
programs were accredited by the Higher Learning Commission and North Central Association of 
Schools (HLC/NCA). Both universities had recently redesigned their general education programs to 
realign with institutional culture and disciplinary diversity, and need for breadth of knowledge. Hence 
the two universities broadly shared the same philosophy and goals for general education. However 
their approaches to curriculum and assessment as relating to CT were unique. 

The general education program at South-Western University was governed by two committees. 
One committee oversaw the curriculum and the other assessment of general education outcomes. With 
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regard to general education, the university had courses specifically designated as teaching for CT. It 
was assumed that faculty taught for or focused on the development of CT in such designated course. 
The assessment office at the university extracted student artifacts from courses designated as teaching 
CT. Over each summer, a multi-disciplinary panel of faculty was recruited and paid to rate the student 
artifacts for CT outside the context of the student grade for that course. For example: a student 
submitted a response for a class assignment for the course Biology 101. The professor would grade the 
assignment which contributed to the student’s grade for that course. After that, the assessment office 
would make a copy of the artifact, remove any identifier information and hire a different faculty to 
grade the artifact specifically for CT. Raters used a discipline-general rubric developed at the 
university to assess for CT in all student artifacts and writing was the only type of student artifact used 
for the assessment of CT. For purposes of VSA accountability, the university used the standardized test 
called Proficiency Profile from ETS which also assessed for discipline-general CT. 

At Mid-Western University, the general education program was described as foundational to 
the university’s academic plan and integral to a liberal education with a goal to create life-long 
learners. Critical thinking was given a place of importance as an outcome of general education. A 
General Education Coordinating Committee which had representation from several colleges and units 
within the university provided leadership to the general education program. With regard to critical 
thinking, the university used a discipline-general rubric to assess a specific student rather than a 
specific artifact. As a response to the VSA, Mid-Western had piloted the use of the CLA, a 
standardized, discipline-general test. It is currently in the process of collecting data on critical thinking 
for the first cohort as part of VSA requirements to present value-added learning gains in undergraduate 


students. 
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Sampling 

This study employed a combination of stratified sampling (Miles & Huberman, 1994; Patton 
2002) and homogenous sampling (Miles & Huberman, 1994). Disciplines identified as general 
education disciplines (Allen, 2006) were clustered as represented in the table below: 


Discipline Cluster 

Disciplines 

Humanities 

English, Philosophy, Fine Arts 

Natural Sciences 

Chemistry, Physics, Biology 

Social Sciences 

Sociology, Economics, Psychology 


The sample consisted of 17 faculty members who met these criteria: (a) Teach/Taught general 
education courses in the Humanities, Natural Sciences and Social Sciences, (b) tenured (for individual 
interviews) and tenured and non-tenured faculty (for group discussion), (c) currently teaching in a 
public university. The sample consisted of seven female and 10 male participants. The teaching 
experience of participants, when combined, averaged approximately 11 years. The racial composition 
of the sample consisted of 10 Caucasians, one African American, three Asian Chinese, one Asian 
Indian, one Latino, and one participant from the Middle East. 

Data Collection 

Nine tenured faculty members (3 from each discipline cluster) were interviewed individually. 
Interviews were conducted using a semi-structured interview guide. Questions revolved around (a) 
assessment methods/artifacts used (b) attributes of critical thinking that were assessed (c) assessment 
instruments (d) assessment approach (e) engagement with assessment of CT in general education. Each 
interview lasted an hour, was audio recorded and transcribed verbatim. 

Data was also collected using focus group methodology (Krueger & Casey, 2000). A focus 
group comprising of eight faculty members (different from individual interview participants) drawn 
from disciplines in the social sciences, natural sciences and humanities met for approximately an hour. 


We discussed their understanding of CT, assignments/instruments they had developed to assess for CT, 
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and the challenges and possibilities that their approaches yielded. The focus group methodology 
allowed for multiplicity of perspective (Morgan, 1996) and helped unearth and corroborate patterns 
(Cohen & Crabtree, 2006) observed in individual interviews. 

Participants were also requested to complete a brief demographic profile and survey. The 
survey collected data on faculty engagement with CT beyond their individual courses. It provided data 
to compare and contrast the findings from faculty approaches to CT in the classroom with institutional 
approaches. Participants were asked to share prompts for course assignments, rubrics or other 
instruments that they developed or used to assess for CT in their general education courses. Data from 
these multiple sources provided insights into how faculty approached the assessment of CT and 
allowed for triangulation among data sources. 

Data Analysis 

We determined that the coding processes used by Glaser and Strauss (1967) would best achieve 
our stated goals for data analysis. Transcribed and other data sources were consolidated, coded and 
analyzed using NVivo 9.0. We utilized three coding processes: open coding, axial coding, and 
selective coding (Glaser & Strauss, 1967). The goal during the open coding phase was to, as Patton 
(2002) put it, break apart the data and delineate the larger themes and concepts. Thereafter, axial 
coding (Glaser & Strauss, 1967) was used to assemble the coded data around new categories. As 
suggested by Strauss and Corbin (1990), we brought data together using “conditions, context, 
action/interactional strategies and consequences” (p. 96). The emergent concepts derived through axial 
coding were selectively coded (Strauss & Corbin, 1990) to develop findings on how faculty assessed 
CT. 

Trustworthiness 

The study used several strategies recommended by Maxwell (1996); Miles and Huberman 
(1994) and Johnson (1997) to bolster the trustworthiness of (a) the researchers, (b) the nature and 
process of data collection and, (c) how data was analyzed and interpreted. Strategies like establishing 
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reflexivity, triangulation among data sources, multiple sampling techniques, maintaining the validity of 
description, interlaced with the use of qualitative approaches for data analysis, are all intended to lend 
credibility to findings. 

Findings 

The findings of this study are structured around four broad themes that emerged from the data - 
types artifacts used to assess for CT, instruments used to assess for CT, nature of assignments that 
faculty developed to assess for CT, and faculty awareness of institutional methods used to assess for 
CT. 

Faculty Used Diverse Disciplinary Artifacts to Assess for CT 

Faculty assessed for CT in assignments developed to assess for disciplinary content and 
context. The representation in Table 1 shows the type of artifacts that faculty used to asses for CT in 
their disciplinary courses. Faculty from biology, chemistry and physics used artifacts that would have 
students evaluate knowledge claims using empirical and statistical means. The botanist thought that 
drawing a concept from a particular perspective could help with assessing CT. Faculty from 
philosophy preferred the use of artifacts like quizzes and multiple choice tests that tested for skill and 
mastery of a process. Faculty from the humanities preferred artifacts like portfolios, essays and art 
shows. They held it was important to give students an opportunity to discuss their work and make their 
thinking evident. Faculty from the social sciences used case studies and written assignments to assess 
for CT. 

A closer look at these lists revealed that faulty used an array of artifacts ranging from simple 
multiple choice tests and quizzes, to more elaborate approaches like collaboratively developed 
assignments, drawing, portfolios and displays of work where students were asked to share their 
thinking processes and reasoning. In general, faculty mostly preferred to use pieces of writing of 
significant size to evaluate for critical thinking in students. Faculty did not use a single artifact or end 
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product to assess for CT. They used a combination of formative and summative forms of assessment 
and combined them with observation, listening to, and challenging students’ thought processes. 

Table 1 

Artifacts preferred by faculty to assess for CT 


Natural Sciences 



Chemistry 

Physics 

Botany/Biology 

Analysis of published research 
Observation of students 
Multiple choice tests 

Analysis of published research 
Capstone project 

Analysis of published research 
Multiple choice tests 
Drawing 

Humanities 



Philosophy/Ethics 

Romance Languages 

Studio Arts/Graphic Design 

Quizzes 

Exams 

Writing 

Observation of thinking 
processes 

Essay exams 
Collaboratively designed 
writing assignments 

Presentations 
Pieces of writing 
Talk about the work 
Hear them and listen and 
respond 

Watch them create 
Work with them 
Combination of craft and 
concept 

Art shows of the final work 
Portfolios 

Social Sciences 



Economics 

Sociology 

Psychology 

Written assignments 
Case studies 
Classroom discussions 

Written assignments 
Critiquing movies 
Field work 

Written assignments 
Classroom discussion 
Evaluation of current research 
Verbal debate assignments 


While the above table uses disciplinary clusters to present data, the approaches preferred by 
faculty to assess for CT could not be easily consolidated under discipline clusters. While faculty from 
chemistry, physics, psychology and philosophy were more comfortable with identifying specific 


summative artifacts for the assessment of CT. Faculty from biology, romance languages, sociology, 
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studio arts and design identified formative artifacts with which they associated elaborate 
developmental processes. For some faculty, CT was expressed as a complex, developmental process 
rather than an artifact. In their opinion, a fair assessment of CT needed to consider the effort that went 
into decision making, the process of idea generation, ideas that emerged during the process of making 
of a piece of craft or art, self correction, and responses to feedback. The informant from studio arts 
said, “I can hear them and listen to them and, um, I guess respond to them, too, and know if they’re 
growing.” Each of these stages, he argued, contained elements of CT. As a result, CT was more 
assessable in their disciplines during the developmental process than in the final product. 

Some participants used rich imagery to capture the CT process. An informant from romance 
languages said, “I don’t want a mirror. I want their movie, but they actually have to direct and write 
that movie for me to have it.” An informant from studio arts described CT as represented in 
assignments as a process that “bakes and grows.” The nature of the assignments they identified 
involved a process that led to the final product. Speaking of a craft portfolio as an end product, he said, 
“We’re looking at their portfolio at the end. And so I wouldn’t be able to assess that (CT) necessarily at 
that point in time.” Faculty in the humanities argued that the end product by itself was not sufficient to 
communicate the depth of the critical thought that went into the creation of these artifacts. 

Faculty Rejected the use of Multiple Choice Tests as a Valid Measure of CT 

Though we did not specifically ask a question about the use of multiple choice questions to 
assess for CT, most faculty brought it up in their interviews with a lot of passion. Faculty from 
disciplines like chemistry, physics, philosophy and biology who held to more rationalistic or formulaic 
conceptualizations of CT appeared more inclined to tolerate the use of multiple choice tests and 
quizzes. However, they all highlighted the limitations of multiple choice tests and Scantron bubbles to 
assess for CT. A professor from philosophy held “oftentimes we end up doing quizzes and exams, and 
things like that are not as good at getting at the core skills. . .but they could capture whether you had the 
right truth at the end.” A participant from botany spoke of her experience using multiple choice and 
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Scantron bubbles when she said, “They might not always show you what you want. . .1 don’t think it's 
assessing what they really know.” A faculty from chemistry admitted he used multiple choice tests but 
rued, “Yeah, I don’t know if multiple choice is good for critical thinking.” 

A scathing rejection of multiple choice and Scantron bubble type tests came from faculty in the 
humanities and social sciences. A professor of romance languages argued, “Perhaps the vilest 
quantification I can imagine is the infamous, multiple choice exams. Fill in the blank or, uh, you know, 
block in the number or fill in the box with, back and forth, back and forth. I don’t think that that's a 
valid measure.” The professor of philosophy argued, “I guess multiple choice tests couldn’t capture the 
richness of the critical thinking process. . .but they are a hell of a lot better than Scantron bubble sheets 
for getting at core skills.” The psychologist argued “multiple choice tests, I think they couldn’t capture 
the richness of the process.” Given this finding, the continued use of standardized, multiple choice tests 
to assess for CT in general education or for accountability purposes has implications for gaining 
faculty buy-in for such assessment methods or its outcomes. 

Faculty Did Not Explicitly Assess or Measure CT in Students 

Interestingly, faculty in this study did not use or develop any instrument to assess for CT 
explicitly within student artifacts. When asked to produce rubrics or instruments that they used to 
assess CT, their responses are best captured in their own words: 

Interviewer: Do you use any instruments like rubrics or tests to assess for CT in your courses? 
Chemistry: “No. Well, not at the undergraduate level.” 

Philosophy: “No, we have no formal rubric, it depends on the student. 

Physics: “We know it when we see it.” 

Studio Arts: “I can show you one that the director of assessment gave me. It’s in a file 
somewhere.” 


Psychology: “I do not.” 
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Economics: “Frankly speaking, I have that somewhat in my head and, uh, I do not, um, 

I have that in my head.” 

Biology: “I think my TAs use a rubric.” 

Romance languages: “No I don’t use one.” 

As is evident, none of the faculty who participated in this study was able to produce rubrics or 
any assessment instrument used to explicitly assess for CT in classroom assignments. As a result, 
faculty did not have a way of knowing whether students had progressed as critical thinkers in their 
courses. The professor from chemistry epitomized this response: 

Interviewer: But then you have no way to assess for CT in your classroom, right? 

Chemistry: Right. 

Interviewer: You don't know whether you're being effective on that count. 

Chemistry: Oh, heavens no. I have no idea whatsoever. You know... 

This finding leads logically to the question, how do faculty assess for CT? 

Faculty Took a Faceted, Implicit and Disciplinary Approach to Assess CT 

The finding that faculty do not use formal assessment measures like rubrics to assess for CT 
must not be construed that they do not assess for CT. During the interviews, my attempt to interpret 
their lack of formal assessment approaches as a lack of assessment of CT was met with stiff resistance. 
For instance, I asked the professor from philosophy, if it would be fair to say that he used an “I know it 
when I see it approach” to assessing CT. He retorted, “Well I would say mostly yes to that, with a 
reminder that we are regularly teaching them specific notions of validity and soundness and so when 
we know it when we see it; it’s not porn.” He continued, “We’re looking for good reasoning and we’re 
looking for them to engage in these reasons of validity and soundness.” The informant from physics 
argued, “Do we have implicit rubrics in our head, I think we do. Do we ever write them down? No.” 
Such responses were typical of participants in all disciplines. They all claimed to assess for CT but 
implicitly using core skills and learning outcomes that are woven into their disciplinary content. An 
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analysis of some assignment prompts that faculty used in courses they teach will help strengthen this 
finding. 

Samples of assignment prompts used in the humanities based disciplines 


Course: French Literature 

Freedom, opportunity, the future. Trace the parallel development of these hopes in events, ideas 
and personalities in the French Revolution and the Caribbean slave revolts. Did either or both 
succeed, explain. 

Course: Spanish Civilization 

What elements (religious, political, social, economic) made the Spanish Habsburg empire a 
surprise? Given those elements, was the empire actually inevitable in light of late medieval 
European state growth? Explain. 

Figure 1. Sample assignment prompts used in Romance Languages. 

Course: Introduction to Philosophy 

Is the following argument valid and sound? Explain your answers. 

Premise 1 : If the moon is made of cheese then it is made of a dairy product. 

Premise2: The moon is not made of a dairy product. 

Conclusion: Therefore, the moon is not made of cheese. 

Course: Introduction to Philosophy 

Descartes says such and such. Evaluate what Descartes said, explain and evaluate his reasoning 
and arguments in terms of validity and soundness. 

Course: Ethics and Morality 

Is abortion right or wrong? Objectively evaluate whether the reasons provided by both pro life and 
pro choice stances are good reasons for their conclusion; whether you agree with them or not. 

Figure 2. Sample assignment prompts used in Philosophy. 

A closer examination and comparison of the assignment prompts from the humanities revealed 

that they were heavily situated in perspective and used the language of words to explain and establish 

meaning, thereby providing a vehicle for the expression of critical thought. However, there were 

disciplinary distinctions to be marked within the humanities. The assignments in philosophy (Figure 2) 

sought to formalize the rules of logic where sound premises lead to logical conclusions or truth value. 

They also clearly provided students with the criteria for the judgment of ideas — rationality, validity 


and soundness. This assessment approach is in line with the procedural, formulaic and logical 
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conceptualization used for instruction of CT in philosophy. The assignments in ethics provided 
nuanced differences from philosophy. It required the evaluation of two competing views on abortion 
with equally valid arguments or logical merit. CT was assessed as the ability to see the arguments on 
both sides and possibly conclude that there was no definitive evidence to take a stand on the issue. 

Marking distinctions among sub-disciplines in philosophy, the philosopher remarked, “It 
should be said that, there are lots of subparts of philosophy and its various parts are more or less 
affected by formalizing or mathematizing reasoning but the core areas of philosophy and various 
secondary areas like philosophy of mind, philosophy of science; those are very much inflected by the 
mathamatization or formalization logic. It’s less evident in ethics or aesthetics but only somewhat 
less.” As a result he said that in philosophy, the use of quizzes and competitions, as to who could arrive 
at the accurate conclusion first was a good means to assess for CT. He pondered, “But you know, how 
would I compare or rank the skills of some famous ethicists against the skills of some famous 
philosopher of science. I mean what would you do. . .much less to try and rank them against a mechanic 
or an architect.” While his statement showed distinctions even within sub disciplines within a 
discipline. It also revealed how, in his mind, forms of assessment of CT were deeply rooted within 
specific disciplinary contexts. 

The assignment from romance languages (Figure 1) sought an explanation of thoughts and 
perspectives on literary and historical texts. Interestingly, the participant from romance languages was 
also a historian. She admitted, “I approach this (CT) from two angles - one, from romance language 
because this is where I am, but secondarily, I’m a historian. So I look at it in terms of analytical 
thought, specifically, persuasive, rigorous analytical thought usually leading to a structured completion 
or point of view.” As a literary expert, she argued that students needed to use both historical and 
literary texts to mark their answers. She referred to it as “multi-sourcing” to answer a historical 
question. She argued that in her dual disciplinary role, she tended to see historical texts as “artifacts, or 
expression, but seldom as indigenous to the time and people who produced them.” Clearly, she was 
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among the group of historians who saw history as belonging to the humanities rather than to the social 
sciences. The nature of the assignments she developed and her expectations for the assignments 
emerged from her epistemic, disciplinary stance; as did the criteria she used to assess for CT in such 
artifacts. 

The assignments from literature and studio arts focused on the individual narrative, sought to 
establish contextual meaning and recognized genre. The informant from studio arts captured this 
approach well when he argued, “everyone has a kind of a bent, visually - a visual kind of bent that 
relates to some aspect of the field. Maybe it's a traditional format, maybe it’s very avant-garde. Maybe 
it’s just working with stones and gold or, you know, whatever. And so based on their particular 
interests, I would guide them and offer them suggestions or ideas to get them started.” Faculty from 
languages, studio arts and design said they looked for the application of ideas, depth of analysis and 
breadth of research. While some wanted their students to reach dialogic conclusions or an expression 
of or the resolution of ideas, others thought CT led to acquiring an extra dimension of awareness. 

Samples of assignment prompts used in natural science disciplines 


Course: Introduction to Biology 

Read the following published article and write an essay using the following questions: 

1. What problem or question is the author trying to address? 

2. Does the author state the problem clearly? 

3. What is the author’s hypothesis in the article? 

4. Does the author make a sound argument in which the facts logically support his position? 
Course: Introduction to Botany 

If you were to look on the earth, what would it look like if you're an ant? Draw your ideas from 
an ant's perspective. Contrast an ant's perspective to the perspective of a man on the moon. 
Figure 3. Sample assignment prompts used in Biology. 


Course: Organic Chemistry 

Molecules interact with other molecules and affect the very physical properties of those 
molecules. Use this information to predict how a mixture of these two materials might interact - 
ethanol and CFRMgBr. 

Course: Genercd Chemistry 
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Design an experiment to test the following hypothesis: 

Figure 4. Sample assignment prompts used in Chemistry. 

As is evident from Figures 3 and 4, faculty in the natural sciences generally focused on 
assignments that involved empiricism. They focused on problem solving, scientific reasoning, 
synthesis of ideas and the testing of causal hypotheses. The informant from Chemistry who held that 
CT involved the synthesis of information, said, “I'd like to include questions that I call, essentially 
synthesis questions where they have to take independent facts and come up with another hypothesis.” 
Informant from biology, who conceptualized CT as problem solving and decision making said she 
used the following criteria: “Can they state the hypothesis? Did they come to a logical conclusion? The 
conclusion may be different, but is it logical based on the knowledge that they have?” The assignments 
in the natural sciences focused on assessing for the ability of students to evaluate knowledge as 
represented in published research articles and to examine the logic, reasoning and conclusions 
presented by an author using logical reasoning and statistical validity as evaluative criteria. 

Samples of assignment prompts used in social science disciplines 

In the social sciences, the assignment prompts that faculty developed showed that they valued 
the analysis and evaluation of competing views and multiple perspectives. Interestingly, within the 
social sciences the language through which critical thought was expressed played a huge role in the 
evaluation of the outcome of CT. Faculty approached CT depending on their methodological 
persuasion. 

Course: Introduction to Psychology 

Read the following published article. Dr. Smith found a relationship between eating a lot of fish 
and living longer, he wants everyone to eat lots of fish, what do you think about that 
conclusion. 


Figure 5. Sample assignment prompts used in psychology. 
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Course: Introduction to Sociology 

Pick a department store in two specific locations. You are to visit each and observe the store 
using unobtrusive measures (see your text). Your report will include your field notes on each 
store as well as your analysis of your observations. In your observations you should report at a 
minimum on the following: 

1. Your perceptions of the store based on the outside: 

Does it seem safe? Is there ample parking? Would you shop here? 

2. Your perceptions of the store based on the inside: 

Is the store clean? Are the shelves stocked? Are there any store personnel accessible? 

Is the produce fresh? 

3. A comparison of both stores: 

Are there any differences between the two stores? Does one store seem to have better 
service than the other? Do store personnel seem more helpful to patrons in one store 
compared to the other? 

4. Sociological Analysis - How might we explain these differences using one of the three 
theoretical perspectives discussed in chapter 6-7 of the text? What are some sociological 
implications of your findings? 


Figure 6. Sample assignment prompts used in sociology. 

Despite the emphasis on empiricism, faculty in the social sciences acknowledged the role of 
perspective in CT as evident in Figure 6. CT involved developing a substantiated conclusion. The 
informant from economics held, “as long as they are thinking and bringing in different perspectives, 
and they could come up with answers or assumptions, that I completely do not agree with, but they are 
based on whatever information they had, or whatever website or source they looked at. . .based on that, 
if they've come up with a conclusion and they have thought through it and put it in the model and made 
some predictions, I'm fine with that.” The quantitative psychologist said she looked for “the ability to 
synthesize and integrate previous literature. The conclusions have to be logical, tied back to the 
introductory material.” CT to them involved students reaching a substantiated conclusion through 
empirical means. 

Considering the assessment prompts and approaches used by faculty to assess for CT in the 
humanities, social and natural sciences, there is strong evidence to show that they developed 
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assignments using their disciplinary conceptualizations of the term. The skill sets that each of these 
faculty identified and assessed for, were robustly disciplinary with sufficient nuance even among sub 
disciplines in a field. This finding raises serious questions on the efficacy of approaches that use 
questions or assignments that are devoid of context or set within contexts that are alien to some 
disciplines, or assess for a standard set of discipline-general CT skills in all students irrespective of 
discipline. CT when applied to real life situations or disciplinary contexts is not understood by faculty 
as a procedural set of steps or standardized skills. Faculty viewed CT as faceted and used a rich 
conglomeration of approaches and disciplinary artifacts to assess CT which cannot be captured or 
assessed using a single, discipline-general standardized test or rubric. 

Faculty Assessed for CT as Integrated with Other Learning Outcomes 

This study found that faculty assessed for CT as integrated with disciplinary content and other 
learning outcomes. Faculty held that for CT to be evident in student work, it required forms of 
expression like writing, oral presentations, drawing, craft or art. The professor of philosophy argued, 
“What we are looking for when we talk about critical thinking skills is the ability to comprehend, 
articulate, analyze, and then, of course, for yourself produced this kind of premise-conclusion 
argument.” The professor of romance languages said, “There are individuals who are capable 
obviously of great critical thinking, but they cannot readily offer it up to other people.” 

At the same time, faculty held that other learning outcomes like writing or oral communication 
were enhanced through the use of CT. One informant captured this connection well when she said, 
“Communication is worthless unless you assess a certain value to it because all of it is not equal. All of 
it is not the same.” The professor of philosophy held “Students do pretty well at reasoning and not 
pretty good at talking about what’s working in their reasoning and, of course, that makes them bad at 
extending those skills to other areas.” The presence of CT in disciplinary artifacts enhanced the value 
of an artifact and the value of other learning outcomes. 
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Faculty from economics, philosophy, physics and romance languages, also argued that the 
expression of critical thought is intrinsically tied with a student’s ability in the preferred language of 
expression (numbers or words) for critical thought in their disciplines. The economist said, “I am 
looking for critical thinking, but that’s not the only thing that I'm looking for.” Professor of physics 
said “In science, mathematics is the equivalent of our language. . . You need to know how to manipulate 
this language of mathematics.” He continued, “We never look at just critical thinking in isolation. 
We're also looking whether they gained disciplinary skills.” An informant from Chemistry held, “A 
large portion of the undergraduate education is learning to use the language and to apply it, to know 
when to use it, to know when to use the right kind of skills.” The participant from romance languages 
argued “If you have a great idea and you cannot express it in any language, either human or social or 
numerical, you have a problem.” When assessing for CT in their disciplinary courses faculty factored 
in the ability of students in the means of communication, the ability to express oneself in the preferred 
language for critical thought in the discipline and the integration of disciplinary content, CT and other 
learning outcomes. 

Faculty had Little Awareness of Institutional Approaches to Assess CT in Gen Ed. 

Faculty pointed to logistical problems in connecting with the mission and formal assessment 
practices of the general education program. Faculty from disciplines like philosophy and biology 
argued that individual professors cannot take credit for the development of CT skills in students nor 
can they be blamed for the lack of it in students. The difficulty of placing responsibility for the 
development of CT as an outcome of general education is complex as responsibility does not rest with 
a single faculty member, department or college. The general education program transcends specific 
courses, departments and colleges; making the assessment process and paths for accountability 
complex and ill defined. In addition, faculty pointed out that the tenure of an academic quarter or 
semester makes it difficult for specific faculty to adequately focus on the overall development of CT in 
students. Faculty in the natural sciences rued that they needed to focus on memorization and learning 
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of disciplinary content and by the time they were ready to focus on teaching for CT, the term of the 
semester had expired. A few faculty complained of time constraints and large class sizes for lack of 
focus on CT. 

In addition, all of them admitted that they had not been formally trained to teach or assess for 
CT as an outcome in their courses nor did they know the role they or their courses played in the larger 
general education program. All faculty participants responded that they did not know if an operational 
conceptualization existed at the college or university level. All of the participants said that their 
departments did not have an operational conceptualization of CT. It must be pointed out that both 
universities at which this study was conducted used specific conceptualizations of CT in their general 
education programs and used standardized tests to assess for CT as part of the VSA accountability. 

The study also found that faculty disconnected from their own approaches for assessing for CT 
when called upon to explicitly assess for CT. Even though faculty from different disciplines developed 
elaborate assignments and processes to assess for CT through their disciplinary assignments, when 
asked about how they would assess for CT explicitly if they had an opportunity to do so, faculty said 
that they would use the discipline-general, rubric developed by the University for General Education. 
An informant from physics said, “I’d use a rubric that I based off of the gen ed one.” The following 
exchange with a faculty from studio arts shows the danger of the unquestioned use of existing rubrics 
or tests by faculty to formally assess for CT: 

Interviewer: So if I were a university officially charged with, assessing critical thinking 
development in students, and I came up, and I said, you need to help me assess critical 
thinking for your department or your program, what would you do? 

Studio Arts: I would go over here to my assessment book, and we've actually been 
working on this. 

We discovered while looking at the rubric in his file, the components he attributed to CT were 
not contained in the rubric. His reasons for the disconnect are captured in this exchange: 
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Interviewer: Would you say you're kind of doing a disservice there, this rubric assesses 
just the conceptual. You’re relegating CT to a purely cognitive skill, whereas we’ve 
spoken about this richness of critical thinking in your discipline that’s beyond cognition. 

But yet when you assess, you’ve resorted to a more rationalistic kind of approach. 

Studio Arts: Well I think you would find if you were in attendance at these things that, 
um, . . . that, um, you know, the discussions are pretty candid and, um, you know our 
hand is forced really to have to make descriptions on scenarios of people. And critical 
thinking, I see that as something that happens when they’re in the classroom working 
with the instructor. Where ideas are emerging and developing. I don’t think we’re 
looking at the beginning of where the critical thinking ability is. 

The pressure that he and his colleagues faced in conforming to formal discipline-general 
assessment of CT is borne out in the words “our hand is forced really to make descriptions on 
scenarios of people.” His formal assessment approach looked for aspects of CT that he criticized as 
meaningless and was seeking to assess for CT at a stage in the development of the artifact when it was 
least evident. This clearly showed a misalignment between faculty and institutional approaches in the 
assessment of CT. 

Both universities from which data was collected in this study used discipline-general rubrics 
and summative forms of assessment to assess CT in general education. What is disturbing is that 
faculty did not question why they needed to use the general education rubric or how it reflected their 
discipline or classroom approaches. The danger with marketing tests or rubrics titled “critical 
thinking”, without conceptualizations of CT and the intent and purpose for which such instruments 
were developed is evident. Labels like “Critical Thinking Rubric” lulls even questioning disciplinary 
experts into believing the instrument can assess for CT in their disciplinary contexts. 
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Discussion/Implications 

A Multi-disciplinary Vs a Discipline-General Approach to Assess CT 

This study has offered empirical evidence that faculty took a robustly disciplinary approach to 
assessing CT. It revealed that they assessed for CT differently using multiple approaches, methods, 
epistemologies and contexts across disciplines. Their approaches showed elements of uniqueness even 
within sub-disciplines in a field. Further, within specific courses, faculty took a faceted approach 
assessing for CT using preferred languages for the expression of critical thought. They generally 
focused on facets of CT that were relevant to the content being taught in a course. This raises questions 
on the validity of the prevailing assumption in general education that CT is discipline-general and the 
use of discipline-general rubrics and tests to assess for CT in all students irrespective of discipline. 

The faceted approach that faculty took to CT calls for efforts to consolidate the curriculum and 
assessment of CT at the program level. There is a need for developing curriculum maps in general 
education that identify the courses that teach for CT, the specific facets of CT that faculty focus in the 
course content and the epistemology that guides CT in that discipline. It is also imperative that such 
maps indicate the developmental level in which CT is focused on. Such an approach can help bring 
together the multiple facets of CT through program integration; thereby providing a valid reflection 
and consolidation of faculty efforts at the course level. 

It must not be forgotten that the goal of a liberal education and general education in particular 
is to give students the widest spectrum of disciplinary exposure. If that is true, the value of a liberal 
general education lies not in the generality of CT but its multidisciplinarity. Hence we recommend a 
complex, disciplinary, faceted approach to assessing CT and an expansion of the scope of CT to reflect 
the multidisciplinarity of the term rather than its generality. Such an approach will ensure that students 
experience and are assessed on a wider range of critical thinking capabilities and dispositions. Such an 
approach can represent the aspirations of multiple disciplines, be a valid representation of the critical 
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thinking capabilities of students, reflect faculty approaches and help fulfill the promises of a liberal 
education. 

Assessment of CT should factor competency in the medium of communication 

Faculty across disciplines heavily relied on writing as the basis to assess CT. Faculty in 
disciplines like biology, psychology, philosophy and fine arts disciplines preferred the use of diverse 
mediums of expression to assess for CT. These approaches are not represented in general education 
programs and much less in standardized assessment approaches. When one considers this finding and 
research on multiple intelligences (Gardner, 1985; Sternberg, 2010), there is a need to diversify the 
type of artifacts used to assess for CT in general education. 

Another implication of this finding is that any interpretation of the results of assessing critical 
thinking should factor in the ability of a student to communicate in the medium through which critical 
thought is expressed. Most universities tend to assess and treat the two outcomes separately. While the 
skills may be assessed separately, it is important to factor in writing ability of students when 
interpreting critical thinking scores. 

A Contextualized Vs Standardized Approach to Assessment 

As is evident in current approaches used by general education programs and the VS A, we have 
traded quality for expediency and complexity for simplicity. Given that faculty across disciplines 
rejected or acknowledged the ineffectiveness of multiple choice tests as a valid measure of CT, the use 
of such tests at the institutional or national level raises serious questions on the validity of such 
measures. In addition, such assessment methods do not reflect how faculty assess for CT in the 
classroom. Without establishing the validity that faculty seek in assessment measures; standardized 
forms of assessment do little to gamer faculty buy-in for assessment or their confidence in the 
outcomes of assessment. 

The contextualized, faceted approach that faculty adopted, repudiates the validity of using 
universal approaches to assess for CT. It also highlights the meaninglessness of using a misaligned 
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model to assess for CT at the institutional level. Thereby raising questions on the inferences and 
conclusions drawn based on these tests; about student ability, or faculty and institutional efforts to 
develop critical thinking in students. 

Need to make the Implicit, Explicit 

Faculty largely approached the teaching and assessment of CT implicitly through disciplinary 
content and artifacts. In addition, faculty engaged their personal epistemologies to make judgments on 
CT. There are multiple underlying currents that are called into play when faculty think about or are 
called on to teach or assess for CT. The process of making those assumptions and connections overt 
and bringing those assumptions from the subliminal to the consciousness was powerful during the 
interview process. Faculty were re-evaluating their ideas, aligning and realigning their thoughts. 

Several times faculty used phrases like “I don’t know” “I am not sure” “is that correct?” in the 
interviews. 

The implicit disciplinary approach coupled with the “I know it when I see it approach” is not 
particularly student friendly. A key assumption of assessment done for general education and 
accountability purposes is that CT can be extracted and assessed as a distinct outcome. Hence, we 
argue that there is a need for faculty to focus making the implicit, explicit through the use of learning 
outcomes, curriculum maps and publishing the criteria they use to assess for CT. 

Implications for VS A Efforts to Assess CT at the National Level 

The findings of this study have implications for the VSA’s efforts to assess CT among 
participant institutions in the United States. First, we must seek to understand the conceptualizations 
and frameworks that underlie the usage of the term CT, in the standardized tests recommended by the 
VS A. A definition or test of CT represents a school or particular interpretation that was developed for a 
specific purpose. It would be naive to assume that the term critical thinking as used in the three tests is 
comparable. The universal application of such approaches without an acknowledgement of the context 
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or purpose for which conceptualizations or tests were developed can be dangerous and damaging to the 
field given the complexity of usage that this study has uncovered. 

In a timely study, Stassen, Herrington and Henderson (2011) compared their institutional 
definition of CT with those that underlie the three VSA recommended standardized tests and the 
VALUE rubric. They found little in common between their university’s definition of CT and the 
definitions used in the three VSA recommended standardized tests. In their analysis they found that the 
VALUE rubric most closely aligned with their campus definition. However they listed as a limitation 
of their study, that their campus definition did not “reflect faculty beliefs about the relative importance 
of each of these constructs” (p. 134). Their study strengthens our findings that faculty and their 
approaches to assessing CT are being ignored in the VSA’s approach. 

To recall, the study by Escoe, Hall and Nicholas (2009) showed no correlation between the 
CLA (a VSA recommended test) and faculty approaches to assess for CT. Our examination of faculty 
approaches to assessing CT has revealed some reasons as to why faculty approaches to assessing CT 
may not correlate with results produced by national, standardized, discipline-general tests. First, within 
the context of the classroom, faculty taught for CT implicitly through disciplinary content and assessed 
for it implicitly through disciplinary assignments. In addition, outcomes like CT are expressed in the 
preferred language for the expression of critical thought in a discipline. CT is further expressed 
through other learning outcomes like written, oral communication, craft or portfolios. Hence, the 
proficiency of student in a disciplinary method, preferred language for the expression of critical 
thought and medium of communication which are factored into faculty assessment are not reflected in 
standardized tests or are measured exclusively using writing as in the CLA or through reading in the 
Proficiency Profile. Faculty in their assessment practices and approach did not see CT as particularly 
extractable from these contexts. However, most standardized tests and summative forms of assessment 
assume that CT is a distinguishable, separate skill that can be defined, extracted and assessed 


independently. 
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Second, this study found that faculty across disciplines used multiple approaches to assess for 
CT. The most popular approach used significant pieces of student writing. Approaches used by faculty 
in some disciplines in the social sciences and humanities are not captured by methods used in 
standardized tests. While faculty from disciplines in the natural sciences and philosophy were more 
comfortable with identifying specific artifacts for the assessment of CT, faculty from biology, romance 
languages and fine arts struggled to answer the question. The nature of the assignments they identified 
involved a process that led up to the final product. CT was more assessable during the process than in 
the final product. 

Third, as this study found, the nature of disciplinary content, more so in disciplines in the 
humanities and some social sciences was given to perception and multiple layers of context. This 
brought into play the personal epistemologies of faculty into their judgments of CT. Standardized tests 
contain or narrow the contextual elements of CT. As a result, these tests do not capture interpretations 
of CT that are meaningful to faculty and their approaches and the judgments they make of CT as 
disciplinary experts. 

Summary 

The exploratory, qualitative approach revealed a disconnect between assessment approaches 
used by faculty and approaches used at the institutional level and with approaches used for 
accountability purposes at the national level. In the clamor to define, measure and assess CT, and to 
hold some entity within the university system accountable for the development of CT in students 
faculty as a demographic has been ignored. It highlights the need to engage faculty in all stages of the 
assessment process. If faculty knowledge of their own conception of CT or their assessment 
approaches holds no weight, and is ignored or disrespected, then assessment will remain a bureaucratic 
task with no relevance to the classroom. This study has provided insights that can help with aligning 
institutional approaches to assess for CT with approaches used by faculty. 
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As is evident from this study, the contested nature of critical thinking as a concept, its complex 
interaction with disciplinary knowledge, the epistemology of the thinker and the assessor of thinking 
calls for a complex approach to pedagogy, curriculum and assessment. The risk in using standardized 
approaches to assess CT is that the epistemic, ontological and methodological assumptions that 
underlie such instruments may not align with how CT is operationalized in the classroom or 
incorporated into general education curriculum or assessment by faculty. The continued use of such 
standardized instruments becomes a futile exercise that provides no real value in assessing CT or 
students’ development as critical thinkers. This study reiterates that the assessment of CT in general 
education should adopt complex and multiple assessments. While it is difficult to identify the shape 
that new assessment models to critical thinking should take, a simplistic response is one that higher 
education can ill afford in this age of accountability. 
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